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ABSTRACT 



In the reporting of individual student results from standardized tests 
in educational assessments, the percentile rank of the individual 
student is a major, if not the most prominent, numerical indicator. For 
example, in the 1998 and 1999 California Standardized Testing and 
Reporting (STAR) program using the Stanford Achievement Test 
Series, Ninth Edition, Form T (Stanford 9), the 1998 Home Report and 
1999 Parent Report feature solely the National Grade Percentile 
Ranks. (These percentile rank scores also featured in the more 
extensive Student Report). This paper develops a formulation and 
presents calculations to examine the properties of year-1, year-2 
comparisons using these individual percentile rank scores. The 
approach and formulation follows the previous investigations of the 
accuracy of the individual percentile rank score in Rogosa (1999a). A 
typical question that this paper addresses is: What are the chances 
that a student who really improved 10 percentile points from year-1 
(1998) to year-2 (1999) obtains a lower percentile rank in year-2 than 
year-1? Such questions are addressed using the test reliability 
coefficient in classical test theory to represent quality of measurement. 
Thus we can investigate the question. What level of test reliability is 
needed to obtain good accuracy in year-1, year-2 comparisons? 



1. Technical Formulation 



The technical formulation is a basic errors-in-variables model with all components 
having Gaussian distributions. As in Rogpsa (1999a), all that is meant by the 
phrase “classical test theory calculation” is to identify the calculations as pertaining 
to the simplest case of constant error variance across the score distribution with 
continuous, Normally distributed scores. The components of what is referred to as 
the classical test theory calculation are listed below; use the subscript i = 1,2 to 
indicate properties of year- 1 or year-2 measurements. 

• The cumulative distribution function of the observed scores Y in the national 

norming sample is denoted by Gj(Y) for year-1 or year-2. The classical test theory 
formulation defines this norming distribution, with density function gj(Y), to be a 
Normal Distribution; denote the corresponding population mean and standard 
deviation for Y by fo*’ year-1 or year-2. 

• The observed measure Y contains error of measurement e . The classical test 

theory assumptions dictate that the error of measurement, denoted by e, has a 
Normal Distribution with mean 0 and constant variance across the score 
distribution: i.e., e - N(0, ). (More general formulations, such as 

depending on the level of the test score, can be incorporated into many of these 
results, with the overhead of added complexity.) 

• The test reliability coefficient is often used as an index of the quality of 
measurement. The test reliability is defined for the full (norms^ population; from 
the classical test theory formulation, the reliability is reZj = (ct^^ - )/CTn^ . For a 
rough, but useful, illustration set the reliability of a 60-item test to be .90 (in line 
with standardized achievement tests). Then use Spearman-Brown to obtain the 
rough test length equivalents for various reliability values: 

reliability .60 .65 .70 .75 .80 .85 .90 .95 

number items 10 12 16 20 27 38 60 127 

• The norming distributions, Gj(Y), are based on fallible Y-scores. An alternative 

is to consider what the norming distribution would be if measurement had been 
perfect (i.e., not distorted by error of measurement in Y). At the risk of over- 
complicating the notation, denote by Gj(Y) the cumulative distribution function 
with corresponding mean and standard deviation (p^i > [(^Ni ~ )] ^)! ^iOO 

represents a (hypothetical) norming distribution not distorted by measurement 
error (i.e., constructed from scores with reliability 1). 

• The score for an individual student examinee is denoted by S, The percentile 
rank (PR) for the score S is 100 Gj(S); thus Gj(S) can be thought of as a 
nondecreasing transformation of the score S to the percentile rank metric. The score 
S has underlying true score T ; the measurement model is S = T + e . An individual 
under perfect measurement has percentile rank in observed norming distribution 
100 Gj(t) or, in a norming distribution not distorted by measurement error, the 
percentile rank is 100 G*(t). Often in the calculations, an individual (or an 
individual's achievement level) is characterized a value of Gj(t) . 



2. Accuracy of Year-to-Year Improvement in Percentile Rank Scores 

The observed improvement is 02(82) - Gj(Si), the signed difference between the 
percentile rank scores for year-2 and year-1; improvement may be positive or 
negative. The main accuracy calculation is for the quantity: 

yly2 = Pr{ 02(82) - Gj(Si) < bound | Gi(Ti), G2(t2)} , ( 1) 

the probability that the improvement in the percentile rank scores is less than or 
equal to the quantity "bound" (bound may be negative or positive) for a student with 
stated year-1 and year-2 values Gj(Tj) and G2 (t2). 

2.1 Computation of yly2 Probability: Technical Details 

Preliminaries. Let 0[x] indicate the distribution function (cdf) for N(0,1) and (t>[x] 
indicate the density (pdf) for N(0,1)- Then G-(x) = 0[(x - |XNi)/ONj], and 
8j I T- ~ N[ Ti , - rel/^ ] so that Pr{ 8^ < x} = 0[(x - - rel-) ' Also note 

that T- = UNi + <'^rel-) 0 ”^[G*(t)] and Of Gj(t) + p] = ^ ^^(t) + p]. 

The computation of the yly2 probability is implemented using the following 
conditioning argument. For a student having a specified value for Oi(Ti)^^condition 
on a draw of an Sj from the 81-distribution (8 i|Ti ~ N[ Tj, 0^1(1 - relj) ] ) and 
express that 81-value in terms of its fractile of the Si-distribution, psj , to obtain: 

Pr{ 02(82) - Gi (81) < bound I pSi } = Pr{ 82 < G2 “'[Gi(8i) - f- bound] | pSi } = 

0[{0”^[0[(1 - reli)^" + (Vreli) Gi(Ti)] ] + bound] - 

(Vrel2) 0 -'[G;(t 2)] }/(1 - rel2)‘^“] . (2) 

As a side note to (2), Gi(8i) can be expressed in terms of psj as: 

Gi(8i) = 0[(1 - reli)'''^ 0"'[psi] + (Vreli) G;(Ti)] ] . 

Then uncondition (2) by integrating Pr{ 02(82) - Gi(8i) < bound | psj } over pSj in 
[0,1]: 

1 

yly2 = J [o[{0'^[0[(l - reli)*'^' ^'^[psi] + (Vreli) Gi(Ti)] ] -f- bound] - 
® (Vrey 0 )-'[G;(t 2)] }/(1 - rey'^'^jldpsi 



( 3 ) 



2.2 Calculations and Illustrations 



Maintaining Percentile Rank, G*(Ti) = G 2 (t 2 ). Table 1 displays values of yly2 in (3) 
for a student who has maintained percentile rank in year-1 and year-2 in the sense 
of G*(Ti) is set equal to G 2 (t 2 ). The entries in Table 1 also use the simplification of 
(3) in setting the year-1 and year-2 test reliability coefficients to be equal, relj = 
rel 2 - Consequences of different reliabilities are also discussed below. 

Table 1 presents values of yly2 in (3) for test reliability values from .70 to .95 (.7, 

.8, .85, .9, .925, .95). For each reliability value the rows of each sub-table represent 
a student's value of G*(Ti) = G 2 (t 2 ), so that the .60 row indicates a student who 
"really belongs" at the 60“’ percentile in both year-1 and year-2. For a test with 
reliability .90 administered in year-1 and year-2, that student has a 11.6% chance 
of showing a decrease of at least 20 percentile points and also a 11.6% chance of 
showing an increase of at least 20 percentile points. 

To compare these accuracy results for year-1, year-2 comparisons in Table 1 with 
more traditional assessments of uncertainty, consider values of the standard error 
of G 2 (S 2 > - Gi(Si). The results for s.e.[G 2 (S 2 ) - Gi(Si)] are obtained from derivations 
of the moments of the percentile rank score in Rogosa (1999c). For boJ;h year-1 and 
year-2 tests having reliability .90, s.e.[G 2 (S 2 ) - Gj(Sj)] is 0.1702 for Gj(Tj) = G 2 (t 2 ) = 
.50. Increase the test reliabilities to .95 and this s.e.[G 2 (S 2 ) - Gj(Sj)] becomes 
0.1231. 

Insert Table 1 here 

Another version of the statement about the student with Gj(Tj) = G 2 (t 2 ), = .60 and 
test reliability .90 is that the probability is .768 that the magnitude of the observed 
change, | G 2 (S 2 ) - GjCSj) 1 , is less than .20. Figure 1 displays these type of 
probability statements by plotting the values of Pr{ G 2 (S 2 ) - Gi(Sj)^ < bound} - 
Pr{ G 2 (S 2 )-Gi(Si) < -bound} as a function of test reliability for Gj(Tj) = G 2 (t 2 ) = 
.50 and .75 or .25. 



Insert Figure 1 here 

The result for yly2 in (3) allows the year-1 and year-2 test reliability coefficients, 
relj and rel 2 to differ. The consequences of different test reliabilities can be charted 
in various ways. Rather than be exhaustive, some specific examples are considered 
here. With G*(Tj) = G 2 (t 2 ) = .50, the effect of differing test reliabilities is minimal in 
the following set-up. Taking the base as relj = rel 2 = .90, the difference in yly2 
values between the base (i.e.. Table 1 entries) and differing reliabilities relj = .85, 
rel 2 = .95 is less than .001 for the listed values of bound. The same result is found 
for comparing relj = rel 2 = .80 with relj = .75, rel 2 =^.85. Moving away from Gj(Xj) = 
G 2 (t 2 ) = .50, differences do emerge. Take Gj(Tj) = G 2 (t 2 ) = .75, then the 



Table 1. 



Pr{G 2 (S 2 > - Gi(Si) < bound I G*(Ti), G 2 (t 2 )} for reliability .70 to .95 and G*(Ti) = G 2 (t 2 >. 



Reliability .70 



bound 



- .20 


- .15 


-.10 


-.05 


0.0 


.05 


. 10 


.15 


.2 


.10 0.124 


0.183 


0.263 


0.369 


0.5 


0.631 


0.737 


0.817 


0.876 


.20 0.186 


0.248 


0.323 


0.408 


0.5 


0.592 


0.677 


0.752 


0.814 


.30 0.219 


0.279 


0.348 


0.422 


0.5 


0.578 


0.652 


0.721 


0.781 


.40 0.235 


0.294 


0.359 


0.428 


0.5 


0.572 


0.641 


0.706 


0.765 


.50 0.24 


0.299 


0.363 


0.43 


0.5 


0.57 


0.637 


0.701 


0.76 


.60 0.235 


0.294 


0.359 


0.428 


0.5 


0.572 


0.641 


0.706 


0.765 


.70 0.219 


0.279 


0.348 


0.422 


0.5 


0.578 


0.652 


0.721 


0.781 


.80 0.186 


0.248 


0.323 


0.408 


0.5 


0.592 


0.677 


0.752 


0.814 


.90 0.124 


0.183 


0.263 


0.369 


0.5 


0.631 


0.737 


0.817 


0.876 


Reliability .80 


bound 














- .20 


- .15 


-.10 


- .05 


0.0 


.05 


.10 


.15 


.2 


.10 0.072 


0.126 


0.211 


0.337 


0.5 


0.663 


0.789 


0.874 


0.928 


.20 0.137 


0.202 


0.287 


0.388 


0.5 


0.612 


0.713 


0.798 


0.863 


.30 0.174 


0.241 


0.319 


0.407 


0.5 


0.593 


0.681 


0.759 


0.826 


.40 0.194 


0.259 


0.333 


0.415 


0.5 


0.585 


0.667 


0.741 


0.806 


.50 0.2 


0.265 


0.338 


0.417 


0.5 


0.583 


0.662 


0.735 


0.8 


.60 0.194 


0.259 


0.333 


0.415 


0.5 


0.585 


0.667 


0.741 


0.806 


.70 0.174 


0.241 


0.319 


0.407 


0.5 


0.593 


0.681 


0.759 


0.826 


.80 0.137 


0.202 


0.287 


0.388 


0.5 


0.612 


0.713 


0.798 


0.863 


.90 0.072 


0.126 


0.211 


0.337 


0.5 


0.663 


0.789 


0.874 


0.928 


Reliability .85 


bound 














- .20 


- .15 


-.10 


- .05 


0.0 


.05 


.10 


.15 


.2 


G ^ di ) 

.10 0.043 


0.089 


0.173 


0.311 


0.5 


0.689 


0.827 


0.911 


0.957 


.20 0.103 


0.168 


0.258 


0.371 


0.5 


0.629 


0.742 


0.832 


0.897 


.30 0‘.141 


0.21 


0.295 


0.394 


0.5 


0.606 


0.705 


0.79 


0.859 


.40 0.162 


0.23 


0.312 


0.403 


0.5 


0.597 


0.688 


0.77 


0.838 


.50 0.168 


0.237 


0.317 


0.406 


0.5 


0.594 


0.683 


0.763 


0.832 


.60 0.162 


0.23 


0.312 


0.403 


0.5 


0.597 


0.688 


0.77 


0.838 


.70 0.141 


0.21 


0.295 


0.3 94 


0.5 


0.606 


0.705 


0.79 


0.859 


.80 0.103 


0.168 


0.258 


0.371 


0.5 


0.629 


0.742 


0.832 


0.897 


.90 0.043 


0.089 


0.173 


0.311 


0.5 


0.689 


0 . 827 


0.911 


0.957 


Reliability .90 


bound 














- .20 


-.15 


-.10 


- .05 


0.0 


.05 


.10 


.15 


.2 


G ;( Ti ) 

.10 0.017 


0.047 


0.121 


0.27 


0.5 


0.73 


0.879 


0.953 


0.983 


.20 0.06 


0.119 


0.213 


0 . 344 


0.5 


0.656 


0.787 


0.881 


0.94 


.30 0.096 


0.163 


0.256 


0.372 


0.5 


0 . 628 


0.744 


0.837 


0.904 


.40 0.116 


0.186 


0.276 


0.383 


0.5 


0.617 


0.724 


0.814 


0.884 


.50 0.123 


0.193 


0.282 


0.387 


0.5 


0.613 


0.718 


0 . 807 


0.877 


.60 0.116 


0.186 


0.276 


0.383 


0.5 


0.617 


0.724 


0.814 


0.884 


.70 0.096 


0.163 


0.256 


0.372 


0.5 


0.628 


0.744 


0.837 


0 . 904 


.80 0.06 


0.119 


0.213 


0.344 


0.5 


0.656 


0.787 


0.881 


0.94 


.90 0.017 


0.047 


0.121 


0.27 


0.5 


0.73 


0.879 


0.953 


0.983 



9 



Reliability .925 



bound 



. 10 


-.20 

0.007 


-.15 

0.026 


-.10 

0.087 


.20 


0.037 


0.087 


0.18 


.30 


0.067 


0.13 


0.226 


.40 


0.085 


0.153 


0.248 


.50 


0.091 


0.16 


0.254 


.60 


0.085 


0.153 


0.248 


.70 


0.067 


0.13 


0.226 


.80 


0.037 


0.087 


0.18 


.90 


0.007 


0.026 


0.087 



Reliability .95 



bound 



Gi*(xi) 


- .20 


-.15 


- . 10 


.10 


0.001 


0.008 


0.047 


.20 


0.014 


0.048 


0.131 


.30 


0.033 


0.085 


0.18 


.40 


0.047 


0.106 


0.203 


.50 


0.052 


0.113 


0.21 


.60 


0.047 


0.106 


0.203 


.70 


0.033 


0.085 


0.18 


.80 


0.014 


0.048 


0.131 


.90 


0.001 


0.008 


0.047 







- .05 


0.0 


.05 


.10 


.15 


.2 


0.238 


0.5 


0.762 


0.913 


0.974 


0.993 


0.322 


0.5 


0.678 


0.82 


0.913 


0.963 


0.3 53 


0.5 


0.647 


0.774 


0.87 


0.933 


0.367 


0.5 


0.633 


0.752 


0.847 


0.915 


0.371 


0.5 


0.629 


0.746 


0.84 


0.909 


0.367 


0.5 


0.633 


0.752 


0.847 


0.915 


0.3 53 


0.5 


0.647 


0.774 


0.87 


0.933 


0.322 


0.5 


0.678 


0.82 


0.913 


0.963 


0.238 


0.5 


0.762 


0.913 


0.974 


0.993 



- .05 


0.0 


.05 


. 10 


.15 


.2 


0.19 


0.5 


0.81 


0.953 


0.992 


0.999 


0.286 


0.5 


0 . 714 


0.869 


0.952 


0.986 


0.323 


0.5 


0.677 


0.82 


0.915 


0.967 


0.339 


0.5 


0.661 


0.797 


0.894 


0.953 


0.344 


0.5 


0.656 


0.79 


0.887 


0.948 


0.339 


0.5 


0.661 


0.797 


0.894 


0.953 


0.323 


0.5 


0 . 677 


0.82 


0.915 


0 . 967 


0.286 


0.5 


0 . 714 


0.869 


0.952 


0.986 


0.19 


0.5 


0.81 


0.953 


0.992 


0.999 



6 

10 





Figure 1. Plots of Pr{ 02(82) - Gi(Si) < bound} - Pr{ 02(82) - Gi(Si) < -bounc^ as 
a function of test reliabdity for G*(ti) = 02(^2) = .50 in top frame and Gt(xi) = G2(”C2) 
= . 75 or . 25 in bottom frame for labeled values of bound = {. 05 , . 10 , . 20 ). 
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differences in yly2 values for relj = relg = .80 are larger by .02 to .03 than yly2 values 
for relj = .75, relg = .85 and smaller by .02 to .03 than yly2 values for relj = .85, relg = 
.75 . But the two-sided probability statement used in Figure 1, Pr{ 02(82) - Gj(Si) < 
bound} - Pr{ 02(82) - Gi(8i) < -bound}, changes less than .002 for bound = .10 and 
with relj = rel2 = .80 compared to relj = .9, rel2 = .7 or relj = .7, re^ = .9. With 
bound = .20, the change in yly2 values is less than .001 for these different reliability 
configurations. Calculations for year-1, year-2 comparisons based on an actual 
standardized achievement test, 8tanford 9, and in which the tests have different 
reliabilities (and different norms distributions) can be found in Rogosa (1999b). 

Increasing Percentile Rank, G*(Ti) .10 = G2(t 2). Table 2 displays values of yly2 in 
(3) for a student who improved percentile rank 10 points from year-1 to year-2 in the 
sense of G2 (t2) = G*(Ti) . 10. The entries in Table 2 also use the simplification of (3) 
in setting the year-1 and year-2 test reliability coefficients to be equal. One basic 
question Table 2 informs about is. What's the probability of seeing a decline in the 
observed percentile rank, even when the student has made a noticeable improvement? 
(by setting bound = 0 in Table 2). For G*(Ti) = .4 or .5, that probability is .36 for test 
reliability .8, .295 for test reliability .9 and .217 for test reliability .95. 

Insert Table 2 here 

These probability statements, Pr{ 02(82) - Gi(8i) < bound }, such as in Table 2 allow 
a decline in scores to result from Gi(8i) being "too high" (Gi(8i) > Gi(Xi)) as much 
from 02(82) being much "too low." Another view of these kind of calculations can be 
obtained from Equation (2) by setting psj = .5, which results in a fixing of the value of 
Gi(8i)= Gi(Ti)= 3>[Vreli) G*(Ti)] ]. Thus 02(82) is the only random component 
in the student improvement. And for the simplest comparison take Gi(Xi) = .50, as 
that results in Gi(8i) = Gi(Xi) = G*(Xi) = .50. By fixing pSj = .5 a large random 
component of 02(82) - Gi(8i) is removed, and thus we would expect a quantity such as 
Pr{ 02(82) - Gi(8i) < 0} given a "true" increase G2(x2) = Gi(Xi) + .10 would become 
smaller than the results shown in Table 2 (which are obtained from E^quation 3). 
Below is a comparison of Pr{ 02(82) - Gi(8i) < 0} given G2(X2) = .6, Gj(Xj) =.5 from 
Equation (2) (fixing Gi(8i) = .5) and Equation (3). The Equation (2) quantities are 
smaller, but perhaps not by as much as expected (especially for the lower reliability 
values). 
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Table 2. Pr{ 02 ( 82 ) - Gi(Si) < bound | G*(Ti), GgCTg)} for reliability .80 to .95 and G*(Ti) + . 10 = G 2 (T 2 ) • 
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Increasing Percentile Rank, Gi(xi) + .20 = G 2 (t 2 ). Table 3 displays values of yly2^in (3) 
for an even larger student improvement from year-1 to year-2 in the sense of G 2 (t 2 ) = 
G*(Ti) -I- .20. The entries in Table 3 also use the simplification of (3) in setting the 
year-1 and year-2 test reliability coefficients to be equal. Again, one question to 
examine is, What's the probability of seeing a decline in the observed percentile rank, 
even when the student has made a noticeable improvement? Setting bound = 0 in 
Table 3, for G*(Xi) = .3, that probability is .229 for test reliability .8, .133 for test 
reliability .9 and .053 for test reliability .95. 



Insert Table 3 here 



Table 3. Pr{ 02 ( 82 ) - Gi(Si) < bound I G*(Xi), for reliability .80 to .95 and Gi(Ti) + .20 - G 2 (t 2 ) • 
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Decreasing Percentile Rank, G*(Xi) - .10 = G 2 (t 2 >. Also, the setting in Table 2 can be 
turned around to examine a student with a "real" decline from year-1 to year-2, in the 
sense of G*(Xi) > G 2 (X 2 ). Table 4 shows values of yly2 in (3) for a student with Gi(Xi) - 
. 10 = G 2 (x 2 >. The entries in Table 4 also use the simplification of (3) in setting the 
year-1 and year-2 test reliability coefficients to be equal. Table 4 shows that even 
with G*(Xi) - .10 = G 2 (x 2 >, the probability of obtaining an increase of 10 or more points 
in observed percentile rank is a large as .132 for test reliability .90. 



Insert Table 4 here 

Guaranteeing positive improvement! Not possible, but it is of interest to ask. How 
much real improvement is needed in order to obtain high probability of an observed 
improvement? The entries in Table 5 set high probability as .90 and ask how large 
k = G 2 (x 2 ) - G*(Xj) needs to be in order for Pr{ G 2 (S 2 ) - Gj(Sj) > 0} = .90. Even for test 
reliability .95, k needs to be as large as .165. 



Insert Table 5 here 



Table 4. 

Pr{ 02 ( 82 ) ”Gi(Si) < bound I G* 
G;(Ti) - .10 = 0*2(12) . 
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Tables. Values ofk such that Pr{ 02 ( 82 ) -Gi(Si) > 0 I Gi(Xi) + k= G 2 (t 2 )} 
reliability .80 to .95. 
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3. Consistency of Percentile Rank Scores Over Years 



It depends what the meaning of "consistency" is. Another approach to describing 
accuracy of the percentile rank scores over successive years is to consider the setting 
in which G*(x^ is the same over two (or more years), e.g., .60 in year-1 and year-2. 
And then ask, given constant G*(Xj) (G*(Ti) = GgCxg)), how consistent are the observed 
Gi(Si) and G 2 (S 2 >? 

In Rogosa (1999a), one approach to describing the accuracy of a percentile rank score 
was to calculate, for a student whose percentile rank under perfect measurement is 

100 ^ ^ 

hit-rate^ = Pr{|Gj(Sj) - Gj(x^ 1 < tolerance | Gj(Xj)}. 

And from Rogosa (1999a), for year i 

hit-rate^ = G*{Tj) -k tol ] - (Vrel^) G*(t^] }/(1 - relA^'] - 

G*(Xj) - tol ] - (Vrelj) 0 "HG*(t^)] }/(1 - rel^) '] . 



And thus a measure of year-to-year consistency is the probability that Gi(Si) is 
within the designated closeness to Gj(Ti) and^ G 2 (S 2 ) is within the designated 
closeness to G 2 (t 2 ) (with typically G/Tj) = G 2 (t 2 )): 

consistency j 2 = hit-rate j hit-rate 2 . 

Table 6 presents values of consistencyj 2 for three values of the tolerance: tol = .01, 
tol=.025, and tol = .05. For example, with a test reliability of .90 for both years, a 
student "really" at the 60^’’ percentile has probability .026 of both observed percentile 
rank scores being within 2.5 percentile points of the 60^’’ percentile (i.e., in the range 
57.5 to 62.5) and probability .101 of both years' observed percentile rank scores being 
within 5 percentile points of the 60^*' percentile. 



Insert Table 6 here 



Table 6. 

Year-1, Year-2 Consistency of Observed Percentile Rank Scores . 
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